System and method for providing a customized graphical user interface based on user inputs

ABSTRACT

A system and method that provides customized graphical user interfaces on mobile devices based on user inputs. An exemplary method includes detecting a computing device remotely connected to a remote server over a network and having an active session of a software application running on the remote server. Moreover, the method further includes identifying and selecting one or more hotkey buttons based on the detected software application, transmitting the one or more hotkey buttons to the computing device to be displayed in a customized interface while the software application is active, detecting an activation of the one or more hotkey buttons displayed on the computing device, and executing, by the remote server, an operation for the active software application in response to the activation of the one or more hotkey button by the user.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority as a continuation ofU.S. patent application Ser. No. 17/483,268 filed Sep. 23, 2021 whichhas issued as U.S. Pat. No. 11,544,088; which itself claims the benefitof priority as a continuation of U.S. patent application Ser. No.15/822,279 filed Nov. 27, 2017 which has issued as U.S. Pat. No.11,169,831; the entire contents of each bring incorporated herein byreference.

TECHNICAL FIELD

The present disclosure generally relates to graphical user interfaces oncomputer displays, and, more particularly, to a system and method forproviding a customized graphical user interface based on user inputs.

BACKGROUND

As mobile devices and other electronic appliances continue to developwith increasingly powerful processing capabilities and dynamictouchscreen displays, the software applications available on thesedevices also continues to get more complex. However, with theseincreasingly complex software applications comes a significantcomplexity in the number of keystrokes and other various inputs forusers to interact with these applications, especially in the context ofelectronic gaming, for example. Moreover, while the processing abilitiescontinue to increase for such mobile devices, the complex sequences ofuser interaction within such software application environments stillconsume significant processing resources.

Therefore, a customized and easy interface is still needed enablingusers to easily and quickly interact with such sophisticated softwareapplications. Moreover, it is desired to provide a system and methodthat minimizes resource consumption by the end client device.

SUMMARY

Accordingly, a system and method is provided a customized graphical userinterface based on user inputs. An exemplary method includes detecting,by a remote server, a computing device remotely connected to the remoteserver over a network and having an active session of a softwareapplication running on the remote server; identifying and selecting, ina hotkey database of the remote server, at least one hotkey button basedon the detected active session of the software application on thecomputing device; transmitting, by the remote server, the identified andselected at least one hotkey button to the computing device to bedisplayed in a customized interface on a display screen thereon inconnection with the active session of the software application;detecting, by the remote server, an activation of the identified anddisplayed at least one hotkey button by a user of the computing device;and executing, by the remote server, an operation for the softwareapplication running on the remote server in response to the activationof the identified and displayed at least one hotkey button by the user.

According to another exemplary aspect, the method includes analyzing, bythe remote server, combinations of input sequences received from aplurality of computing devices running active sessions of the softwareapplication; generating, by the remote server, the at least one hotkeybutton based on respective frequencies of the combinations of inputsequences; and storing, by the remote server, the generated the at leastone hotkey button in the hotkey database.

According to another exemplary aspect, the method includes detecting, bythe remote server, a new computing device having another active sessionof the software application; determining device parameters of the newcomputing device that include a device type and touchscreen inputdimensions of the new computing device; selecting, by the remote server,the at least one hotkey button stored in the hotkey database based onthe device type and touchscreen input dimensions of the new computingdevice; and transmitting, by the remote server, the selected at leastone hotkey button to the new computing device to be presented as acustomized interface on a display screen in connection with the activesession of the software application.

According to another exemplary aspect, the identifying and selecting, bythe remote server, of the at least one hotkey button based on thedetected active session of the software application comprisesidentifying the hotkey button based on device parameters of thecomputing device including a device type and touchscreen inputdimensions of the computing device.

According to another exemplary aspect, the method includes logging, bythe computing device, combinations of input sequences entered by a userof the computing device operating the active session of the softwareapplication based on at least one of keystroke modifiers and functionalkeys selected by the user.

According to another exemplary aspect, the method includes transmitting,by the computing device, the logged combinations of input sequences tothe remote server; and generating, by the remote server, the at leastone hotkey button based on the logged combinations of input sequencesreceived from the computing device.

According to another exemplary aspect, the method includes determining,by the remote server, device parameters of the computing device; andgenerating, by the remote server, the at least one hotkey button basedon the logged combinations of input sequences received from thecomputing device and the determined device parameters.

According to another exemplary aspect, the method includes dynamicallyupdating, by the remote server, the at least one hotkey button selectedfor the active session of the software application based on combinationsof input sequences received from other computing devices running otheractive sessions of the same software application as the computingdevice; and transmitting, by the remote server, the dynamically updatedat least one hotkey button to the computing device to be presented inthe customized interface on the display screen in connection with theactive session of the software application.

According to another exemplary aspect, the method includes logging, bythe computing device, combinations of input sequences entered by theuser of the computing device operating the active session of thesoftware application based on at least one of keystroke modifiers andfunctional keys selected by the user; determining, by the computingdevice, whether one of the logged combination of input sequences exceedsa predetermined threshold; and identifying and displaying, by thecomputing device, in the customized interface on the display screen atleast one hotkey button that corresponds to the logged combination ofinput sequences when the predetermined threshold is exceeded.

According to another exemplary aspect of the method, the identified andselected at least one hotkey button displayed in the customizedinterface on the display screen comprises an expandable menu of aplurality of selectable hotkeys that each correspond to a differentselectable combination of input sequences for the software applicationthat is active on the computing device.

According to another exemplary aspect of the method, the expandable menucomprises a first ring having a plurality of user actions associatedwith the software application and a second ring having the plurality ofselectable hotkeys that is displayed on the customized interface uponthe user selecting at least one of the plurality of the user actions inthe first ring.

In yet another aspect, a system provides a customized graphical userinterface based on user inputs. In this aspect, the system includes ahotkey database configured to store at least one hotkey button for asoftware application; and a remote server with a processor configured todetect a computing device that is remotely connected to the remoteserver over a network and that has an active session of the softwareapplication running on the remote server, identify and select, in thehotkey database, at least one hotkey button based on the detected activesession of the software application on the computing device, transmitthe identified and selected at least one hotkey button to the computingdevice to be displayed in a customized interface on a display screenthereon in connection with the active session of the softwareapplication, detect an activation of the identified and displayed atleast one hotkey button by a user of the computing device, and executean operation for the software application running on the remote serverin response to the activation of the identified and displayed at leastone hotkey button by the user.

The above simplified summary of example aspects serves to provide abasic understanding of the present disclosure. This summary is not anextensive overview of all contemplated aspects, and is intended toneither identify key or critical elements of all aspects nor delineatethe scope of any or all aspects of the present disclosure. Its solepurpose is to present one or more aspects in a simplified form as aprelude to the more detailed description of the disclosure that follows.To the accomplishment of the foregoing, the one or more aspects of thepresent disclosure include the features described and exemplary pointedout in the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated into and constitute apart of this specification, illustrate one or more example aspects ofthe present disclosure and, together with the detailed description,serve to explain their principles and implementations.

FIG. 1 illustrates a block diagram of a system 100 for providing acustomized graphical user interface based on user inputs according to anexemplary aspect.

FIG. 2 illustrates a block diagram of a client device 110 for providinga customized graphical user interface based on user inputs according toan exemplary aspect.

FIG. 3 illustrates a block diagram of a remote server 120 for providinga customized graphical user interface based on user inputs according toan exemplary aspect.

FIGS. 4A and 4B illustrates a flowchart for a method for providing acustomized graphical user interface based on user inputs according to anexemplary aspect.

FIG. 5 illustrates a configurable hotkey button displayed on a userinterface of a client device according to an additional aspect.

FIG. 6 illustrates a block diagram of an example of a general-purposecomputer system (which can be a personal computer or server) on whichthe disclosed system and method can be implemented according to anexample aspect.

DETAILED DESCRIPTION

Various aspects are now described with reference to the drawings,wherein like reference numerals are used to refer to like elementsthroughout. In the following description, for purposes of explanation,numerous specific details are set forth in order to promote a thoroughunderstanding of one or more aspects. It may be evident in some or allinstances, however, that any aspect described below can be practicedwithout adopting the specific design details described below. In otherinstances, well-known structures and devices are shown in block diagramform in order to facilitate description of one or more aspects. Thefollowing presents a simplified summary of one or more aspects in orderto provide a basic understanding of the aspects. This summary is not anextensive overview of all contemplated aspects, and is not intended toidentify key or critical elements of all aspects nor delineate the scopeof any or all aspects.

FIG. 1 illustrates a block diagram of a system 100 for providing acustomized graphical user interface based on user inputs according to anexemplary aspect. As shown, the system 100 is generally comprised of oneor more client devices 110A-110D, a key combination generation andexecution server 120 (hereinafter referred to as “server” or “remoteserver” 120) and a network 130 that communicatively couples the clientdevices 110A-110D to server 120. According to the exemplary aspect, theclient devices 110A-110D are each a mobile computing device, such as asmartphone or tablet, for example. It is well known to those skilled inthe art that such mobile devices typically have smaller display screens(relative to conventional desktop personal computers), and, therefore,it can be frustrating or difficult for many users to quickly enter userinput operations (e.g., keystroke combinations) using these smallerscreens. Thus, as will be readily apparent from the disclosure herein,the exemplary system and method enable the user(s) of each mobile device110A-100D to quickly and easily input user operations using a customizedgraphical user interface provided on the respective display screens ofeach mobile device.

It should also be appreciated that while mobile devices are described asthe client devices 110A to 110D according to the exemplary aspect, theexemplary system and method described herein can be provided for anytype of computing device, such as a desktop, laptop or the like, aswould be appreciated to one skilled in the art. Moreover, while onlyfour client devices 110A-110D are shown according to the exemplarysystem 100, it is noted that the exemplary system and method can beprovided for any number of client devices. Moreover, in one aspect, alarge number of client devices (e.g., 100 or even 1000 or more mobiledevices) provide more analysis points for the server 120 to evaluateuser inputs and generate “hotkeys” as will be described in more detailbelow.

As noted above, each client device 110A-110D of the system 100 isconfigured to communicate data across one or more networks 130 to server120 according to the exemplary aspect. The applicable network 130 can beany network for communicating data and data operations and can includeone or more specific communication systems (not shown) that connect thevarious components of the system 100 by wire, cable, fiber optic, and/orwireless links facilitated by various types of well-known networkelements, such as hubs, switches, routers, and the like. It should beappreciated that the network 130 may employ various well-known protocolsto communicate information amongst the network resources. In one aspect,the network 130 can be part of the Internet or intranet using variouscommunications infrastructure such as Ethernet, Wi-Fi, mobiletelecommunication networks, and the like.

According to the exemplary aspect, each client device 110A to 110D isconfigured to install a client/server application (e.g., Parallels®Remote Application Server Client) on the device that is configured tocommunicate via network 130 with the server 120. In turn, server 120 canbe an application server (e.g., Parallels® Remote Application Server)that, among other things, provides the functions of key combination(i.e., “hotkey”) generation and execution and can operate in acoordinate manner with the client/server application installed on eachclient device 110A-110D, as will be described in more detail below.

In the exemplary aspect, remote server 120 is configured to generate andprovide customized button(s) (referred to as “hotkeys”) that can becustomized and configured to be displayed on each specific client deviceenabling the respective user of the device to perform frequently usedkey combinations using the hotkey presented on the customized interface.For example, in an exemplary aspect, these button(s) may representshortcuts of controls selected for different applications, threads orcombinations of keystrokes, or the like.

In one aspect, the client/server application that is running on a clientdevice (e.g., one or more of client devices 110A-110D) is configured tomaintain a hotkey database of shortcuts received from remote server 120.Thus, during operation, the client/server application can monitor andanalyze the combination of such keystrokes and also measure usagefrequency for the active user application also running on the clientdevice. When the usage frequency of a particular keystroke combinationreaches a certain threshold (which can be configured by the device useror a system administration, for example), the client/server applicationcan query the local hotkey database to determine whether the specificshortcut is already known in the database and can be displayed as asingle on-screen element (i.e., actionable hotkey button) with a humanreadable label and/or icon, which can in turn execute the shortcut as asingle user touch action, for example. This label or icon can bedisplayed automatically in one aspect. In another aspect, the user canbe presented with an option as to whether he or she would like to addthis shortcut as an on-screen button to facilitate further usage (i.e.,for providing the single touch action). Yet further, if the shortcut isnot already known in the database, the user can be presented with anoption of whether he or she would like to create a new on-screenelement, provide it with a customizable name, and the like, which can beadded to the local database and saved for further usage.

In yet a further aspect, these identified frequently used combinationscan be transmitted/posted from the client/server application on themobile device to the remote server 120 by network 130. The server 120 isconfigured to receive these combinations of keystrokes and determinedesired/useful key combinations for each client device based on thestatistics of which shortcuts are used more frequently in the userapplications, for example. When the remote server 120 determines thatthe shortcut usage frequency for each combination reaches a definedthreshold (also configurable by the system administrator, for example),the remote server 120 can generate a suggestion for configuring anew-well known on-screen button (assuming this button has not beenpreviously generated. The newly generated key combination (i.e.,“hotkey”) button can then be transmitted back to client device and addedto the hotkey database, for example. Thus, during subsequent operation,this new hotkey can be presented on the user display as a form of acustomized graphical user interface (“GUI”) using the process describedabove.

Furthermore, in the exemplary aspect, the server 120 may be remotelyrunning the application for one or more of the client devices 110A-110D,such that each client device can run an active session of the remotesoftware application that is primarily executed by the remote server120. Upon receiving a user input of a key combination button on theclient device, the button input/activation can be transmitted back tothe server 120 for execution of the operations associated with thehotkey button. By doing so, the exemplary system 100 can provide highersecurity and hardware independence for the client devices 110A-110Daccording to an exemplary aspect. The details and advantages of thesystem 100 will now be described as follows.

FIG. 2 illustrates a block diagram of a client device 110 for providinga customized graphical user interface based on user inputs according toan exemplary aspect. Client device 110 generally represents any one ofclient devices 110A to 110D described above. In general, client device110 can be any type of mobile device, for example, and will include anoperating system and a central processing unit (“CPU”) 210 provided to,among other things, execute client/server application 220, which asnoted above can be Parallels® Remote Application Server Client, forexample. In this aspect, the client/server application 220 can bedownloaded from server 120 and installed and includes software code(e.g., processor executable instructions) in memory, which may beconfigured to execute/facilitate the key combinations on the customizedGUI of the client device 110 according to the exemplary aspectsdescribed herein.

According to an exemplary aspect, the client/server application 220downloaded on client device 110 is configured to execute a plurality ofmodule, including input collection module 230 and input analysis module240. These modules may be part of client/server application 220 in oneexemplary aspect or separate software component(s) according to anotherexemplary aspect. As used herein, the term “module” refers to a softwareservice or application executed on one or more computers, includingclient devices 110A-110D, components, or arrangement of componentsimplemented using hardware, such as by an application specificintegrated circuit (ASIC) or field-programmable gate array (FPGA), forexample, or as a combination of hardware and software, such as by amicroprocessor system and a set of instructions to implement themodule's functionality, which (while being executed) transform themicroprocessor system into a special-purpose device. A module can alsobe implemented as a combination of the two, with certain functionsfacilitated by hardware alone, and other functions facilitated by acombination of hardware and software. In certain implementations, atleast a portion, and in some cases, all, of a module can be executed onthe processor of a general purpose computer. Accordingly, each modulecan be realized in a variety of suitable configurations, and should notbe limited to any example implementation exemplified herein. Forpurposes of this disclosure below with respect to the exemplaryalgorithms, the disclosure generally refers to the client device 110and/or CPU 210 and/or one of the specific modules as shown to performthe various steps of the algorithm.

As is the case for all mobile devices, client device 110 is generallyconfigured to execute and/or provide a plurality of softwareapplications for the user of the client device 110. For example, asdescribed above, the client/server application 220 can be provided byserver 120, as a remote application server that remotely providessoftware applications (e.g., a remote desktop and remote gamingenvironment, etc.) on client device 110. In this configuration, theserver 120 remotely executes the desired application for the clientdevice 110, which is then transmitted and displayed on the client device110 over network 130, as should be appreciated to one skilled in theart. For example, in one particular aspect, the server 120 can provide a“game mode” that allows a user of client device 110 to conveniently playdesktop games by the customized GUI presented on the display 260 (e.g.,LCD screen) on the client device 110, where the game is running on aremote session by server 120.

In the exemplary aspect, the input collection module 230 is activated byclient/server application 220, for example, and is configured to receivekeyboard and/or mouse inputs from the user of the client device 110running the remote application. Assuming the client device 110 is amobile device, such as an iPhone®, for example, the input collectionmodule 230 is configured to monitor and track/record inputs of theelectronic keyboard (or other movement/touch strokes) input by the uservia the user interface of the application that is presented on display260 of the device 110. In this aspect, the input collection module 230is configured to log the usage of key combinations, and the like, andassociate them with the particular type of client device and also forthe current active application, in which the user of the client device110 is working, playing, etc. In one aspect, these “hotkeys” can betracked when the user selects one or more modifier keys with a regular(i.e., non-modifier) key. Modifier keys can be, for example, one or moreof: “ctrl”, “alt”, “shift”, “altGr”, “win”, “command”, “Fn”, “mediakeys”, or the like. Moreover, the client device 110 can determine thecurrent active application from the software agent installed on theremote server 120, for example.

According to the exemplary aspect, the selection of one of the modifierkeys by the user activates the input collection module 230 to begincollecting the keystroke combinations, for example. In other aspects,the input collection module 230 can continuously monitor any non-letterkeystroke inputs, for example, and log combinations. In one aspect, if auser is playing a game, the input collection module 230 can monitorparticular keystroke sequences of arrow inputs relevant to the game. Inanother aspect, if the user is playing a game, such as a combat game orfirst person shooter game, the collection module 230 can monitorparticular combinations of action strokes required to perform aparticular action in the game. For example, a specific combination ofdirectional swipes on the touchscreen of the display 260 may be requiredto perform a particular fighting combination move in a game. Thecollection module 230 is configured to track the repeated combination ofsuch direction strokes in this aspect.

Moreover, the information collected and logged by the input collectionmodule 230 can be provided to analysis module 240 in the exemplaryaspect. In turn, analysis module 240 is configured to analyse andcompare the collected data, including client device type (e.g., model,and operating software version), specific application(s) running,keystroke combination, and frequency of use. As the analysis module 240detects shortcut combinations, the analysis module 240 can store them aslocal records, where each record contains the shortcut key sequence, atimestamp when the shortcut has been detected, and an active remoteapplication identifier (i.e., which application was running when theuser entered the keystroke combination). Having those records, theanalysis module 240 can also calculate how many times a specifickeystroke combination (i.e., a shortcut) has been activated in a givenactive application during a predefined time frame, which can be definedthe device user and/or by an administrator of the client/serverapplication 220, for example. This calculation can be used to determinea shortcut usage frequency for a given keystroke combination.

As further described above, a hotkey database 270 can be downloaded fromremote server 120 and stored on the client device 110. In one aspect,the downloaded hotkey database 270 can be synchronized with a currenthotkey database maintained by the remote server 120 to receive the bestor optimal hotkeys for a given user application installed on the clientdevice 110. Moreover, when a measured shortcut usage frequency for agiven keystroke combination exceeds a given threshold (e.g., 5 instancesof the keystroke combination in one minute), the analysis module 240 canquery the hotkey database 270 to determine whether the specific shortcutis already known in this local database 270. If so, the CPU 210 cancause a single on-screen element with a human readable label and/or iconthat corresponds to the identified hotkey to be displayed on display 260of the client device 110 while the specific user application is “active”(i.e., currently being operated by the user). This single on-screenbutton can be configured to execute the identified keystroke combinationupon as a single user touch action during use of the active application.In a refinement of this aspect, the user of mobile device 110 can bepresented with an option as to whether he or she would like to add thisshortcut as an on-screen button to facilitate further usage (i.e., thesingle touch action). Yet further, if the shortcut is not already knownin the database, the user can be presented with an option of whether heor she would like to create a new on-screen element, provide it with acustomizable name, and the like, which can be added to the database 270and saved for further usage.

Furthermore, the analysis module 240 is configured to cause thecollected data (both known and/or new keystroke combinations, forexample) to be transmitted to remote server 120 for hotkeyidentification, statistical analysis and/or hotkey generation accordingto an exemplary aspect. For example, the collected data can betransmitted at a particular frequency n, for example, periodically suchas once a day, or every time before closing connection by theclient/server application 220 to the server 120. Moreover, in onerefinement of the exemplary aspect, the analysis module 240 isconfigured to filter the data collected by input collection module 230and only transmitted relevant (i.e., possible hot keys and keystrokecombinations) while not transmitting user inputs that are determined tonot be possible keystroke combinations. For example, if a user is typingan email, these types of user inputs will either not be collected byinput collection module 230 and/or will not be transmitted by analysismodule 240 to server 120. As further shown, client device 110 isconfigured to transmit the collected and analyzed data across thenetwork 130 using any convention I/O device, Interface 280, such as anetwork interface controller or the like.

FIG. 3 illustrates a block diagram of a remote server 120 for providinga customized graphical user interface based on user inputs according toan exemplary aspect. In general, server 120 includes a database 310 ofhotkey buttons, a plurality of modules configured to perform theexemplary algorithms described herein, and I/O interface 360 configuredto communicate with the plurality of client devices 110A-110D overnetwork 130, as described above. Although not shown, it should beappreciated that the server 120 is configured to execute applicationsand the software modules described herein and includes the necessaryhardware and software components that would be known to one skilled inthe art, such as one or more computer processing units, for example.

In one exemplary aspect, the database 310 is configured to store aplurality of custom on-screen controls, such customized buttons forkeyboard stroke combinations, mouse stroke combinations, touchscreencombinations, joysticks movement combinations, and the like. The remoteserver 120 further includes a hotkey selection module 320 that isconfigured to select particular hotkey buttons, from hotkey database310, for example, to be provided to a client device running a particularclient/server application and being run on a particular device. Thesecustomized hotkey buttons can be selected and transmitted to the clientdevice 110 when the remote server 120 detects the mobile device'slaunching/activation of the specific client/server application.Moreover, as described above, the client device 110 can maintain ahotkey database 270 for storing hotkey buttons for the particularapplications installed thereon. In this regard, the remote server 120can synchronize with the client device 110 to identify the active (orfrequently used) applications, for example, and download the relevanthotkeys to be stored in the hotkey database 270.

Referring back to FIG. 2 , the client/server application 220 furtherincludes a hotkey display module 250 configured to receive andcustomized hotkey buttons received from server 120 for the particularapplication. The hotkey display module 250 can cause these buttons to bedisplayed on display 260, for example, when the user is playing a gameapplication, which enables the user to conveniently play the desktopgame that running in a remote session from the remote server 120, usingthe client device 110, which can be an iPhone or iPad, for example.Thus, when the user interacts with these customized buttons (i.e.,activates or clicks the hotkey button displayed on display 260), thoseinteractions are transformed into the desired keys/shortcuts presses,mouse movements/clicks, for the particular application.

It should be appreciated that different game applications requiredifferent set of controls to play conveniently, but games of similargenres (e.g., firs person shooters, RPGs, strategies, simulators, etc.),often use the same or similar set of keyboard and mouse actions to play.In other words, if one shooter has a certain set of controls, thenanother shooter most probably has a very similar set of controls, forexample. In the exemplary aspect, when client device 110 downloads andinstalls the client/server application 220, it can be provided with alimited set of predefined on-screen controls (i.e., “pre-sets” or“pre-set hotkeys”) in the hotkey database 270, so that the user does nothave to setup game controls from the scratch for each and every game. Inthis aspect, the pre-set hotkeys can either be presented automaticallybased on the particular game and/or particular device and operatingsoftware or the user can select these pre-set hotkeys manually.

It should also be appreciated that as the amount of games is huge andcontinues to grow, and the various games can differ in terms of “best”on-screen controls configuration, the remote server 120 can figure beconfigured to add and maintain the pre-set hotkeys centrally at thehotkey database 310, for example. Therefore, in this exemplary aspect,the remote server 120 (e.g., Parallels® cloud or within a serve farm)can store predefined the pre-set hotkeys in hotkey database 310 foron-screen controls for various games/genres. When a user launches a gamevia the client/server application 220, the identification of the gamecan be transmitted to remote server 120. If the game is alreadyregistered with remote server 120, the server 120 can return the bestpre-set hotkeys for the game for hotkey database 270, which can bedetermined from previous game users using the method described herein,for example. In contrast, if this game is not registered yet, the remoteserver 120 can further be configured to identify information about thegame on the Internet, for example, determine the game's genre, and thelike, and return the optimal pre-set hotkeys for the game based onsimilar games registered with the remote server 120, for example.

Thus, according to an exemplary aspect, the pre-set hotkeys can beupdated in the hotkey database 310 when a popular game is released or isabout to be released. In addition, the hotkey database 310 can also beupdated based on users' pre-sets analysis. This analysis is aimed tofind common configuration patterns in pre-sets (e.g., which keys areassigned to on-screen controls, what are the mouse sensitivity settings,etc.) for the same games, and deduce the optimal default pre-set hotkeybuttons for each game (or other application). As a result, when a userof a client device 110 launches a particular game for the first time,the remote server 120 will be informed of thelaunching/activation/installation of the game and will identified the“best known” optimal pre-set hotkeys automatically as described above.Moreover, while the user is playing the particular game (or otherapplication), the input collection module 230 will collect user inputcontrols, for example, if the user customizes the pre-sets, or creates anew pre-set, and have this information sent to remote server 120. Thisinformation will enable remote server 120 to dynamically build andupdate the “best known” pre-set hotkey buttons in the hotkey database310 for subsequent users of the same game (or other application).

As further shown in FIG. 3 , remote server 120 also includes keycombination analysis module 330 that is configured to receive the inputinformation collected by the input collection module 230 of the clientdevice 110 and determine the most frequent combinations of keystrokes todetermine the best key combinations for hotkeys. For example, thisdetermination can be based on frequency of use of the combinations(compared with other combinations), i.e., the shortcut usage frequency,for the specific application and also the specific type of mobile device(which may include the type of input, size of touchscreen, etc.). Theanalysed information that can then be provided to hotkey generationmodule 340 that is configured to generate one or a plurality of hotkeybuttons for the particular application (if the hotkey has not alreadybeen created). These hotkey buttons generated by hotkey generationmodule 340 can be stored as software objects in hotkey database 310.Moreover, in one aspect, the remote server 120 is configured todetermine and provide hotkey buttons back to client device 110 inreal-time (while the application on the device is currently running). Asshown, the hotkey selection module 230 is also coupled to receive theoutput of the hotkey generation module 340. Thus, as hotkey buttons aregenerated by the hotkey generation module 340, these buttons can beaccessed by hotkey selection module 320 and transmitted back to clientdevice 110 to be displayed by hotkey display module 250 in real-time forthe running application according to one exemplary aspect.

Otherwise, the generated hotkey buttons can also be stored as softwareobjects in hotkey database 310 and will include metadata associated eachgenerated button with the particular application (or genre ofapplications, for example), and the particular device. In this aspect,when a different client device subsequently launches the same (or asimilar application), the hotkey selection module 320 can identify andtransmit the optimal hotkey buttons for this new/activated applicationas further described above.

As further described above, the remote server 120 may be remotelyrunning the application for one or more of the client devices 110A-110D.These remotes applications are generally shown as remote applications370, which can be remote desktop applications, remote gamingapplications, or the like. Moreover, during operation, the one or moreclient devices can be configured to run an active session of the remoteapplication 370. As a result, the processing of the software applicationwill be primarily executed by the remote server 120, such that when theuser enters inputs (e.g., keystrokes, mouse clicks, press inputs,hotkeys, etc.), these inputs are transmitted to the remote server 120where they are executed for the remote application 370.

With regard to the exemplary aspect, the remote server 120 furtherincludes a hotkey execution module 350 that is configured to receive theuser's input activation of a hotkey button presented on the customizedGUI on display 260 of the client device 110 during an active session ofthe software application. Upon receiving this activation instruction,the remote server 120 is configured to perform the requested key strokecombination (i.e., the input operation) of the particular remoteapplication 370 (as described above), which is also received by theclient device 110 in real-time during the execution of the softwareapplication by the remote server 120. By doing so, the remote server 120can provide higher security and hardware independence for the clientdevices 110, effectively minimizing CPU processing requirements andresource consumption.

As further described above, the remote server 120 is configured tocontinuously and dynamically build the database 310 of hotkey buttons.In an exemplary aspect, the information (i.e., the keystrokecombinations and frequency) can be collected from a plurality of users.For example, this information can be received and analyzed by keycombination analysis module 330 from a plurality of associated clientdevice, such as, devices in different companies, different countries, orwithin RAS Infrastructure of organization for single company, forexample. Moreover, the received information can be analyzed keycombination analysis module 330 for the frequency of use consideringtypes of applications and type of client devices (other parameters canbe taken into account) to determine the optimal hotkey buttons forclient devices in a particular group, for example. Thus, as describedabove, each new user will be provided with the customized hotkey buttonsthat are optimal for the specific application and for the specificdevice. The information about the current relevant buttons (and/or thebuttons themselves) can be sent to client device 110 with a certainfrequency and suggested use, for example.

FIGS. 4A and 4B illustrates a flowchart for a method 400 for providing acustomized graphical user interface based on user inputs according to anexemplary aspect. As shown, initially at step 405, keystrokecombinations and other user inputs (e.g., touch screen swipes, etc.) canbe logged by input collection modules 230 for one or a plurality ofmobile devices (e.g., client devices 110A-110D). The related data forthese applications can then be identified, which includes identifyingthe specific application and device parameters (e.g., device model,operating system, touchscreen dimensions, input functions, etc.) at step410. The collected information can then be analyzed at step 415 toidentify specific input combinations that are and/or may be preferredkeystroke combinations to be used for customized hotkeys. In theexemplary aspect, this initial analysis can be performed by inputanalysis module 240 of client device 110. In another aspect, theanalysis is performed by key combination analysis module 330 of remoteserver 120. In yet another aspect, the input analysis module 240 ofclient device 110 performs a first pass to filter out meaningless inputsand then key combination analysis module 330 of remote server 120performs the more sophisticated analysis of input combinations todetermine frequency of most used/important keystroke and inputcombinations.

In either case, once the input analysis module 240 of client device 110performs its analysis of the collected data, it is transmitted to server120 on a periodic basis, for example, at step 420. The remote server 120is then configured to generate one or more hotkey buttons at step 425using the algorithms described above. In one exemplary aspect, the keycombination analysis module 330 performs this analysis taking intoaccount collected data from a plurality of devices using thisinformation that can be associated with each other by some specificcriteria, such as country, company, device type, or the like. Finally,as further shown in FIG. 4A, the generate hotkey buttons can be storedas pre-set hotkey combinations in database 310, for example, at step430.

FIG. 4B illustrates the providing and execution environment of thehotkey buttons according to an exemplary aspect. As shown, at step 435,the remote server 120 will detect an application (e.g., a remote gamingapplication) running on a client device 110, for example. In analternative aspect, the remote server 120 may detect an active sessionon the client device 110 of the software application (e.g., the remoteapplication) on the remote server 120.

In either case, the remote server 120 will then confirm at step 440whether the application is known to the server, i.e., whether it haspre-set hotkey combination buttons. If not, the method proceeds to step445 causing the client/server application 220 of the client device tobegin logging input combinations and keystrokes as described above.Otherwise, if the application is known to the server, the remote server120 will further obtain the device parameters at step 450. Theseparameters can include the type of device, operating system, screendimensions, and any other applicable information that can be used todetermine the optimal hotkey combination buttons to customize thedisplay of the client device.

As further shown, at step 455, the remote server 120 will identify andtransmit the hotkey buttons to the client device. For example, based onthe application and device type, the remote server 120 can access hotkeydatabase 310 to identify the one or more optimal buttons to be sent tothe client device. These buttons are then displayed on the client device110 by hotkey display module 250 during the running of the application.During operation, the client/server application 220 will monitor andcheck for input/activation of the customized hotkey buttons on thedisplay 260 of the client device 110 and transmit indication of theactivation to remote serve 120. Finally, at step 465, the remote server120 will perform the operations based on the activated button for theremote application running on the client device 110 according to theexemplary aspect. As noted above, the exemplary method minimizesresource and computing functions by the client device. Moreover, thecustomized GUI provided on each client device 110 enables eachrespective user to have a dynamic and easy experience for the particularapplication (e.g., remote gaming application) with customized hotkeybuttons provide to minimize keystrokes and easily activate commonly usedkeystroke combinations required for the application.

As described above, the exemplary system and method for providingcustomized GUI with hotkey buttons can be implemented for any type ofsoftware applications, such as games, word processing applications,browsers, and the like. Thus, keystroke combinations can be monitoredwhile these applications are active and being used by the user toidentify, display and/or generate new hotkey buttons.

In general, it should be appreciated that the vast majority ofapplications already provide some type of hotkey combinations thatinclude at least one modifier key (e.g., Shift, Alt/Option, Ctrl,Win/Cmd and the like) and/or functional keys (F1, F2, and the like). Asdescribed above, the client/server application 220 (and the relatedmodules) is configured to receive the user input events from a native,third-party, or its own custom keyboard, and to transform these userinputs to key press/release events that can be sent to a remote session.Thus, in the exemplary aspect, when the particular client application isactive, it has full visibility on keystrokes received from the user.

Moreover, it is also known that the modifier and functional keys eitherhave specific, well-known, documented key codes (e.g., if the userutilizes a hardware keyboard attached to the device) or those keys areimplemented as a custom keys by the client application itself, sincenative on-screen iOS/Android keyboards, for example, do not have thosekeys (except for “Shift”) so their press/release events can be detectedby the client software application easily and efficiently. Therefore, inthe exemplary aspect, as keystrokes are processed, the input collectionmodule 230, for example, detects when those “special” keys (e.g.,modifier or functional keys) are pressed and which other keys arepressed/released until the “special” keys are released. When all“special” keys are released, the input analysis module 240 can deducethe shortcut from the sequence of key press/release events recorded(i.e., the keystroke combination) as further described above. Forexample, if the user first presses “Ctrl” and then pressed and released“C,” and then releases “Ctrl,” the shortcut “Ctrl+C” will be detected bythe input analysis module 240. In more complex cases, shortcuts can becomprised from multiple “special” keys, such as “Option+Shift+Command+V”in macOS, for example, which helps to paste a text matching the style ofthe current document. As further described above, these keystrokecombinations can then be compared with hotkey buttons in the hotkeydatabase 270 and/or further transmitted to the remote server for furtherstatistical analysis and hotkey generation as described in detail above.

Moreover, the exemplary aspects described above consider a singleon-screen button that can be displayed on the client device 110 for the“active” application, which enables the user to simply touch (or inputby a mouse, for example) the single button to execute the otherwisecomplicated keystroke combination. Thus, as an example, if the user isworking on a client device with macOS and Microsoft Word is the activeapplication, the user may often use the text selection (e.g.,“Shift+Command+Up Arrow”), while in a Browser application, the user mayfrequently use the hotkey to switch windows in the current program(“Shift+Command+(˜)”). Thus, according to this example, when the useropens the Word document, the button “Text selection” (as a hotkey) canbe accessed from the hotkey database and provided to him as part of thecustomizable display. However, when the user switches to Browser, thebutton “Text selection” will disappear and the new button “Switchwindows” (also accessed from the hotkey database) will be presented tothe user in a dynamic manner, for example. Thus, it should beappreciated that when the input analysis module 240 is analyzing thekeystroke combinations obtained by the input collection module 230 andreferencing them with the hotkey database 270, the input analysis module240 will only be accessing the specific hotkeys in the hotkey database270 for the “active” application in an exemplary aspect.

Moreover, although the exemplary aspect described above contemplates asingle on-screen button (or a plurality of individual buttons displayednext to each other on the display 260) that is configured to execute asingle keystroke combination as a hotkey, the hotkey presented on theuser interface of the client device 110 can also be an expandable menuto present a plurality of related hotkey functions in an alternativeaspect.

In particular, FIG. 5 illustrates a configurable hotkey button displayedon a user interface of a client device according to an additionalaspect. As described above, the system and method described herein isprovided to identify the “active” application running on the clientdevice 110. According to this example, it is contemplated that theactive or working application is a word processing application, such asMicrosoft Word, for example. However, it should be appreciated that suchconfigurable user interface can be provided for any type of userapplication. The exemplary aspect provides a method for presentingalternative configurable options for the software application (e.g.,menu, accelerators, context menu, and the like).

Upon detecting the “active” application, the input analysis module canaccess the hotkey database 270, for example, and display a single (butexpandable) on-screen button, which ultimately provides an adjustablemenu of hotkeys. For example, as shown in FIG. 5 , the first view 505 ofthe user interface (e.g., display 260) displays a first button 512 thatcan be displayed to simply provide the user with the option to access“short cut actions”. According to an exemplary aspect, the menu 505 canbe contextual (i.e., to be a called button) or to appear depending onthe context of the active application. In addition, the dashed box 518Ais provided to illustrate a user's first selection to view the firsttier (or ring) of customized actions for the active application. Thatis, upon selection of the button on the customized interface for “shortcut actions”, the button will be automatically expanded in the display260 to provide a plurality of additional actions associated with theactive application.

In that event, as further shown, a second view 510 is displayed thatincludes a plurality of different types of actions/elements selected forthe active application. In other words, the customized user interfaceprovides a circular representation of the set of actions that providesthe user with fast access to a large number of elements ofaction/sub-actions with a minimum amount of manipulation by the user.According to the example, actions 514A to 514D are shown and correspondto “copy”, “paste”, “cut” and “view”, which are each possible firstactions that can be used in the word processing application as would bereadily appreciated to one skilled in the art. In addition, thecustomizable menu/view 510 is shown according to the example as a halfcircle menu. In this regard, additional view controls 516A and 516B areprovided that enable the user to scroll through other possibleactions/elements (in addition to actions 514A to 514D) in either aclockwise or counterclockwise manner. Of course in a refinement of thisaspect, the menu with the first of actions/elements can be provided as afull circle with a complete listing of available actions. Such decisionsregarding the specific display configurations and dimensions can betailored based on the screen size and device type of the client device110 and/or user preferences, for example.

Furthermore, it should be appreciated that the hotkeys presented in thisfirst tier can be selected and presented using the techniques describedabove. In other words, the remote server 120, for example, can build thecustomizable interface to include actions 514A to 514D based on thestatistics accumulated from a plurality of mobile devices using theapplication as further described above. Thus, the four actions 514A to514D can be selected and displayed as the four most frequently usedactions for the active application.

As further shown, the view or menu 510 presented on the user interfacethat is composed of the elements (i.e., the actions 514A to 514D) can beexpanded into additional sub-items according to another aspect. Thus,the first menu/view 510 provides a single ring-level of selectedactions. When the user selects one of these items, a secondary ring canbe opened/expanded and displayed as an additional view, such as menu520. Thus, in this example, the user has selected the action for “View”514D (the user's selection is denoted by the dashed box 518B), whichexpands the menu to view 520. Again, the view 520 is shown as a halfcircle in order to minimize the size of the customized menu, assumingthe application is working on a Tablet or Smartphone, for example. Thus,this menu can also include additional view controls 524A and 524Benabling the user to scroll the available options for actions in eithera clockwise or counterclockwise direction.

As further shown in the view 520, a plurality of sub-actions relating tothe selected “View” 514D can be displayed. For example, these views caninclude “Split” 522A, “Expand” 522B, “Collapse” 522C, as well asadditional actions “Act 4” 522D, “Act 5” 522E, and “Act 6” 522F.Similarly as described above, the presented actions/elements 522A to522F can be selected based on the accumulated statistics to provide themost frequently used hotkeys corresponding to the most frequently usedkeystroke combinations for the active software application. Thus, theselection of these actions/elements can be determined using theprinciples described above, for example. Based on these actions, theuser can select one of the particular actions (e.g., “Split” 522A) whichexecutes the keystroke combination to split the view of the document,which would otherwise require the user to enter a complicated keystrokecombination (especially in the context of the application beingpresented on a device with only a touchscreen such as a Tablet orSmartphone). Thus, the customizable and expandable menu of hotkeys shownin FIG. 5 provides an alternative and additional technique that enablesthe user to easily and efficiently work with a software application on adevice having limited screen size using the principles described herein.

FIG. 6 illustrates a block diagram of an example of a general-purposecomputer system on which the disclosed system and method can beimplemented according to an example aspect. As shown, a general purposecomputing device is provided in the form of a computer system 20 or thelike including a processing unit 21, a system memory 22, and a systembus 23 that couples various system components including the systemmemory to the processing unit 21. It should be appreciated that computersystem 20 can correspond to client device 110 and/or remote server 120described above, processing unit 21 can correspond to the CPU 210,system memory 22 can correspond to hotkey database 310 according tovarious exemplary aspect.

Moreover, the system bus 23 may be any of several types of busstructures including a memory bus or memory controller, a peripheralbus, and a local bus using any of a variety of bus architectures. Thesystem memory includes read-only memory (ROM) 24 and random accessmemory (RAM) 25. A basic input/output system 26 (BIOS), containing thebasic routines that help transfer information between elements withinthe computer 104, such as during start-up, is stored in ROM 24.

The computer 20 may further include the hard disk drive 27 for readingfrom and writing to a hard disk, a magnetic disk drive 28 for readingfrom or writing to a removable magnetic disk 29, and an optical diskdrive 30 for reading from or writing to a removable optical disk 31 suchas a CD-ROM, DVD-ROM or other optical media. The hard disk drive 27,magnetic disk drive 28, and optical disk drive 30 are connected to thesystem bus 23 by a hard disk drive interface 32, a magnetic disk driveinterface 33, and an optical drive interface 34, respectively. Thedrives and their associated computer-readable media provide non-volatilestorage of computer readable instructions, data structures, programmodules and other data for the computer 20.

A number of program modules may be stored on the hard disk, magneticdisk 29, optical disk 31, ROM 24 or RAM 25, including an operatingsystem 35. The computer 20 includes a file system 36 associated with orincluded within the operating system 35, one or more applicationprograms 37, other program modules 38 and program data 39. A user mayenter commands and information into the computer 20 through inputdevices such as a keyboard 40 (which can correspond to display 260) andpointing device 42. Other input devices (not shown) may include amicrophone, joystick, game pad, satellite dish, scanner or the like.

These and other input devices are often connected to the processing unit21 through a serial port interface 46 that is coupled to the system bus,but may be connected by other interfaces, such as a parallel port, gameport or universal serial bus (USB). A monitor 47 or other type ofdisplay device is also connected to the system bus 23 via an interface,such as a video adapter 48. In addition to the monitor 47, personalcomputers typically include other peripheral output devices (not shown),such as speakers and printers.

The computer 20 may operate in a networked environment using logicalconnections to one or more remote computers 49. The remote computer (orcomputers) 49 may be another computer, a server (e.g., servers 24A to24N), a router, a network PC, a peer device, physical equipment 30and/or other common network node, and typically includes many or all ofthe elements described above relative to the computer 20. The logicalconnections include a network interface 51 and connected to a local areanetwork (i.e., LAN) 51, for example, and/or a wide area network (notshown). Such networking environments are commonplace in offices,enterprise-wide computer networks, Intranets and the Internet. It shouldbe appreciated in one aspect that when computer 20 corresponds to clientdevice 110, remote computer 49 can correspond to remote server 120 andthe network can correspond to network 130.

When used in a LAN networking environment, the computer 20 is connectedto the local network 51 through a network interface or adapter 53. Whenused in a WAN networking environment, the computer 20 typically includesa modem 54 or other means for establishing communications over the widearea network, such as the Internet. Moreover, the modem 54, which may beinternal or external, is connected to the system bus 23 via the serialport interface 46. In a networked environment, program modules depictedrelative to the computer 20, or portions thereof, may be stored in theremote memory storage device. It will be appreciated that the networkconnections shown are exemplary and other means of establishing acommunications link between the computers may be used.

In various aspects, the systems and methods described herein may beimplemented in software, in which the methods may be stored as one ormore instructions or code on a non-volatile computer-readable medium.Computer-readable medium includes data storage. By way of example, andnot limitation, such computer-readable medium can comprise RAM, ROM,EEPROM, CD-ROM, Flash memory or other types of electric, magnetic, oroptical storage medium, or any other medium that can be used to carry orstore desired program code in the form of instructions or datastructures and that can be accessed by a processor of a general purposecomputer.

In the interest of clarity, not all of the routine features of theaspects are disclosed herein. It will be appreciated that in thedevelopment of any actual implementation of the present disclosure,numerous implementation-specific decisions must be made in order toachieve the developer's specific goals, and that these specific goalswill vary for different implementations and different developers. Itwill be appreciated that such a development effort might be complex andtime-consuming, but would nevertheless be a routine undertaking ofengineering for those of ordinary skill in the art having the benefit ofthis disclosure.

Furthermore, it is to be understood that the phraseology or terminologyused herein is for the purpose of description and not of restriction,such that the terminology or phraseology of the present specification isto be interpreted by the skilled in the art in light of the teachingsand guidance presented herein, in combination with the knowledge of theskilled in the relevant art(s). Moreover, it is not intended for anyterm in the specification or claims to be ascribed an uncommon orspecial meaning unless explicitly set forth as such.

The various aspects disclosed herein encompass present and future knownequivalents to the known modules referred to herein by way ofillustration. Moreover, while aspects and applications have been shownand described, it would be apparent to those skilled in the art havingthe benefit of this disclosure that many more modifications thanmentioned above are possible without departing from the inventiveconcepts disclosed herein.

What is claimed is:
 1. A method for providing a customized graphicaluser interface based on user inputs, the method comprising: provisioninga hotkey button to form part of the customized graphical user interface(GUI) to be rendered to a user of a software application upon a clientdevice, wherein the provisioned hotkey button is one of a series ofhotkey buttons rendered within an adjustable menu of hotkeys; renderingof the adjustable menu of hotkeys is established in dependence upon acontext of the software application; the provisioned hotkey button isestablished by an analysis module in execution upon another device whichis configured to: analyse a database of stored established inputs toestablish sequences of inputs made by the user to determine whether ausage frequency of specific sequence of inputs made by the user reacheda threshold where the usage frequency of the specific sequence of inputsis established in dependence upon the timestamps of the instances of theestablished inputs comprising the specific sequence of inputs where thedatabase is generated using data provided by the client device;determine whether an occurrence of the specific sequence of inputsexisted within a hotkey database when the determined usage frequency ofthe specific sequence of inputs made by the user reached the thresholddetermine; and generate the provisioned hotkey button when thedetermined occurrence of the specific sequence of inputs did not existwithin the hotkey database, where the provisioned hotkey buttoncomprises data relating to: the specific sequence of inputs within thehotkey database; a graphical user interface element to be rendered asthe provisioned hotkey button within the customized graphical userinterface; and a human readable label to be rendered in association withthe provisioned hotkey button within the customized graphical userinterface; selection of the graphical user interface element within thecustomized graphical user interface by the user causes the specificsequence of inputs to be executed by the software application; and thedatabase of stored established inputs comprises: inputs made by anotheruser of the client device via one or more user interfaces of the clientdevice; an identity of the software application being used by the userwhen the received inputs are generated; and a timestamp relating to whenthe inputs are made; and each user input of the specific sequence ofuser inputs is selected from the group comprising an input to a keyboardassociated with the client device, a motion upon a touch screeninterface of the client device made by the user, a touch upon the touchscreen interface made by the user, a motion made by the user withrespect to a pointing device associated with the client device, and anaction of the user with respect to the pointing device.